Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(yaml): handle when templates are used as Yaml keys #33361

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

fix(yaml): handle when templates are used as Yaml keys #33361

wants to merge 5 commits into from
+145 −20

Conversation

guillaumededrie
Copy link
Contributor

Changes

This add support for cases where templates are used as key in Yaml files.

Context

See: #18470 (reply in thread)

For example, when having a docker-compose.yaml.j2 file like the following:

---
# {{ ansible_managed }}
version: "3.8"

networks:
  {{ network_name }}:
    external: true

services:
  app:
    image: "<NAME>@<HASH>"
    networks:
      - {{ network_name }}
    command: command
    deploy:
      mode: global
      placement:
        {{ placement | ansible.builtin.to_nice_yaml | indent(10) }}
      labels:
        - "name=value"

Renovate was failing with the following error:

DEBUG: docker-compose.extractPackageFile(<REDACTED>/docker-compose.yml.j2)
DEBUG: Parsing Docker Compose config YAML failed
{
  "err": {
    "message": "Schema error",
    "stack": "ZodError: Schema error\n    at Object.get error [as error] (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/zod/lib/types.js:55:31)\n    at ZodUnion.parse (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/zod/lib/types.js:131:22)\n    at parseSingleYaml (/usr/local/renovate/lib/util/yaml.ts:136:17)\n    at Object.extractPackageFile (/usr/local/renovate/lib/modules/manager/docker-compose/extract.ts:39:29)\n    at extractPackageFile (/usr/local/renovate/lib/modules/manager/index.ts:75:9)\n    at getManagerPackageFiles (/usr/local/renovate/lib/workers/repository/extract/manager-files.ts:58:43)\n    at /usr/local/renovate/lib/workers/repository/extract/index.ts:57:28\n    at async Promise.all (index 1)\n    at extractAllDependencies (/usr/local/renovate/lib/workers/repository/extract/index.ts:54:26)\n    at extract (/usr/local/renovate/lib/workers/repository/process/extract-update.ts:159:28)\n    at extractDependencies (/usr/local/renovate/lib/workers/repository/process/index.ts:158:26)\n    at Object.renovateRepository (/usr/local/renovate/lib/workers/repository/index.ts:71:9)\n    at attributes.repository (/usr/local/renovate/lib/workers/global/index.ts:206:11)\n    at start (/usr/local/renovate/lib/workers/global/index.ts:191:7)\n    at /usr/local/renovate/lib/renovate.ts:19:22",
    "issues": [
      "Expected object, received null",
      "Expected object, received null"
    ]
  }
  "packageFile": "<REDACTED>/docker-compose.yml.j2"
}

Documentation (please check one with an [x])

  • I have updated the documentation, or
  • No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

  • Code inspection only, or
  • Newly added/modified unit tests, or
  • No unit tests but ran on a real repository, or
  • Both unit tests + ran on a real repository

@guillaumededrie
Copy link
Contributor Author

I see a lot of side effect especially for Helm charts here…

I have no problems to invest more time, but maybe it would be better to discuss if the way I'm doing it here is acceptable first.

viceice
viceice requested changes Jan 1, 2025
View reviewed changes
Copy link
Member

@viceice viceice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

failing tests

lib/util/yaml.ts
@@ -140,14 +141,25 @@ export function dump(obj: any, opts?: DumpOptions): string {
return stringify(obj, opts);
}

function getShortHash(data: any): string {
return crypto.createHash('sha256').update(data).digest('hex').substring(0, 7);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we've a hash util function, please use that

lib/modules/manager/docker-compose/__fixtures__/docker-compose.4.yml.j2
@@ -0,0 +1,26 @@
---
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reduce to minimum and inline

@secustor
Copy link
Collaborator

secustor commented Jan 2, 2025

Please open a separate discussions regarding this scenario so we can discuss the solution.
E.g. IMO the YAML parser should be configured to allow null objects.

You can tag me there.

@secustor secustor marked this pull request as draft January 2, 2025 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@viceice viceice viceice requested changes

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@guillaumededrie @secustor @viceice